Auto-encoder pre-training of segmented-memory recurrent neural networks

نویسندگان

  • Stefan Glüge
  • Ronald Böck
  • Andreas Wendemuth
چکیده

The extended Backpropagation Through Time (eBPTT) learning algorithm for Segmented-Memory Recurrent Neural Networks (SMRNNs) yet lacks the ability to reliably learn long-term dependencies. The alternative learning algorithm, extended Real-Time Recurrent Learning (eRTRL), does not suffer this problem but is computational very intensive, such that it is impractical for the training of large networks. The positive results reported with the pre-training of deep neural networks give rise to the hope that SMRNNs could also benefit of a pre-training procedure. In this paper we introduce a layer-local pre-training procedure for SMRNNs. Using the information latching problem as benchmark task, the comparison of random initialised and pre-trained networks shows the beneficial effect of the unsupervised pre-training. It significantly improves the learning of long-term dependencies in the supervised eBPTT training.

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

The Diagnosis of Brucellosis in Rafsanjan City Using Deep Auto-Encoder Neural Networks

Introduction: Brucellosis is considered as one of the most important common infectious diseases between humans and animals. Considering the endemic nature of brucellosis and the existence of numerous reports of human and animal cases of brucellosis in Iran, the incidence of human brucellosis in Rafsanjan city was determined in the last 3 years (2016–2018). The main objective of this study was t...

متن کامل

TimeNet: Pre-trained deep recurrent neural network for time series classification

In the spirit of the tremendous success of deep Convolutional Neural Networks as generic feature extractors from images, we propose Timenet : a multilayered recurrent neural network (RNN) trained in an unsupervised manner to extract features from time series. Fixed-dimensional vector representations or embeddings of variable-length sentences have been shown to be useful for a variety of documen...

متن کامل

Wavelets on Graphs via Deep Learning

An increasing number of applications require processing of signals defined on weighted graphs. While wavelets provide a flexible tool for signal processing in the classical setting of regular domains, the existing graph wavelet constructions are less flexible – they are guided solely by the structure of the underlying graph and do not take directly into consideration the particular class of sig...

متن کامل

The Diagnosis of Brucellosis in Rafsanjan City Using Deep Auto-Encoder Neural Networks

Introduction: Brucellosis is considered as one of the most important common infectious diseases between humans and animals. Considering the endemic nature of brucellosis and the existence of numerous reports of human and animal cases of brucellosis in Iran, the incidence of human brucellosis in Rafsanjan city was determined in the last 3 years (2016–2018). The main objective of this study was t...

متن کامل

Unsupervised Feature Learning With Symmetrically Connected Convolutional Denoising Auto-encoders

Unsupervised pre-training was a critical technique for training deep neural networks years ago. With sufficient labeled data and modern training techniques, it is possible to train very deep neural networks from scratch in a purely supervised manner nowadays. However, unlabeled data is easier to obtain and usually of very large scale. How to make use of them better to help supervised learning i...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2013